feat: Add TimestampNTZType support for casts and unix_timestamp by andygrove · Pull Request #3253 · apache/datafusion-comet

andygrove · 2026-01-23T15:12:21Z

Summary

Add comprehensive support for TimestampNTZType (Timestamp without timezone) wherever TimestampType is currently supported.

Key changes:

Add TimestampNTZType to CometCast.supportedTypes
Support casting TimestampNTZ to Long, String, Date, and Timestamp
Add TimestampNTZ support to unix_timestamp function (no timezone conversion needed)
Add tests for TimestampNTZ casts and temporal expressions

Semantic differences from TimestampType:

unix_timestamp(TimestampNTZ): Simply divides microseconds by 1,000,000 (no timezone adjustment)
Cast to Date: Simple truncation (no timezone adjustment)
Cast to String: Format as local datetime without timezone suffix

Test plan

Run cast tests: ./mvnw test -DwildcardSuites="CometCast"
Run temporal expression tests: ./mvnw test -DwildcardSuites="CometTemporalExpression"
Full test suite

🤖 Generated with Claude Code

The csv_scan.rs file uses types from datafusion_datasource but the dependency was not declared in native/core/Cargo.toml. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…port

Add comprehensive support for TimestampNTZType (Timestamp without timezone) wherever TimestampType is currently supported. Changes: - Add TimestampNTZType to CometCast.supportedTypes - Support casting TimestampNTZ to Long, String, Date, and Timestamp - Add TimestampNTZ support to unix_timestamp function (no timezone conversion) - Add tests for TimestampNTZ casts and temporal expressions TimestampNTZ stores local time without timezone context, so: - unix_timestamp simply divides microseconds by 1,000,000 - Casts to Date use simple truncation (no timezone adjustment) - Casts to String format as local datetime without timezone suffix Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…pport

codecov-commenter · 2026-01-23T17:10:23Z

Codecov Report

❌ Patch coverage is 40.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.90%. Comparing base (f09f8af) to head (38b6c29).
⚠️ Report is 907 commits behind head on main.

Files with missing lines	Patch %	Lines
...scala/org/apache/comet/expressions/CometCast.scala	44.44%	1 Missing and 4 partials ⚠️
...c/main/scala/org/apache/comet/serde/datetime.scala	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3253      +/-   ##
============================================
+ Coverage     56.12%   59.90%   +3.77%     
- Complexity      976     1473     +497     
============================================
  Files           119      175      +56     
  Lines         11743    16172    +4429     
  Branches       2251     2686     +435     
============================================
+ Hits           6591     9688    +3097     
- Misses         4012     5131    +1119     
- Partials       1140     1353     +213

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This commit fixes an issue where date_trunc on timestamp_ntz values could produce incorrect results when the truncation crosses DST boundaries (e.g., truncating a December date to October). The fix modifies as_micros_from_unix_epoch_utc to re-interpret the local datetime in the timezone after truncation, ensuring the correct DST offset is used for the target date. Also updates the test to use a reasonable date range (around year 2024) since chrono-tz has limited support for DST calculations with far-future dates (beyond approximately year 2100). Adds documentation about this known limitation to the compatibility guide. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…test - Remove cast tests for TimestampNTZType to LongType (Spark doesn't support) - Remove cast tests for numeric types and BinaryType to TimestampNTZType (Spark doesn't support these casts) - Fix date_format timestamp_ntz test by using UTC timezone explicitly (Comet interprets timestamp_ntz as UTC during cast, which differs from Spark's session timezone behavior for non-UTC timezones) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…pport

…pport # Conflicts: # docs/source/user-guide/latest/compatibility.md

parthchandra · 2026-02-06T00:19:31Z

What is the expected behavior when data is written to a timestamp ntz field with one session timezone and read by another user with a different session timezone?
Can we update all the unit tests to reflect this condition?

andygrove · 2026-02-13T21:31:21Z

I am moving this to draft until the DataFusion 52 upgrade is merged.

Skip timezone conversion for TimestampNTZ arrays in timestamp_trunc. NTZ values are timezone-independent, so truncation operates directly on naive microsecond values without any timezone resolution, avoiding panics on ambiguous DST fall-back times (e.g. 2024-11-03T01:30:00 in US/Eastern). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

andygrove · 2026-02-13T21:37:26Z

What is the expected behavior when data is written to a timestamp ntz field with one session timezone and read by another user with a different session timezone? Can we update all the unit tests to reflect this condition?

Thanks. I added these tests and that did expose a bug, which is now fixed.

…pport

andygrove · 2026-04-02T17:53:59Z

Merged latest from apache/main and fixed the clippy warning (redundant closure in temporal.rs).

The remaining CI failure (timestamp_ntz - cross-timezone Parquet round-trip) is caused by a panic in timestamp_ntz_to_timestamp when encountering non-existent local times during DST transitions. This is fixed by #3865. Once that PR merges, we can merge main into this branch to pick up the fix.

…pport

Three tests added by this PR assert full native execution via checkSparkAnswerAndOperator, but the underlying expressions legitimately fall back to Spark under the PR's declared scope (casts and unix_timestamp only): - hour/minute/second on TimestampNTZ are marked Incompatible (apache#3180) - date_trunc falls back for non-UTC session timezones (apache#2649) because Catalyst wraps the NTZ child in cast(ts_ntz as timestamp) Use checkSparkAnswer for the fallback paths and retain checkSparkAnswerAndOperator where Comet runs natively (cast, unix_timestamp, date_trunc in UTC).

parthchandra · 2026-04-22T23:28:17Z

@andygrove PTAL at #4039. It is based on this PR, squashed, rebased on main, and with the cast changes removed. (I included one cast test from this PR which was not covered by the ones already there).
We can either merge #4039, or reset this PR to #4039 whichever you prefer.

andygrove · 2026-04-22T23:48:57Z

replaced by #4039

andygrove and others added 3 commits January 23, 2026 07:26

fix: add missing datafusion-datasource dependency

8b22901

The csv_scan.rs file uses types from datafusion_datasource but the dependency was not declared in native/core/Cargo.toml. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Merge branch 'fix-missing-datasource-dep' into feat/timestamp-ntz-sup…

b139c45

…port

andygrove changed the title ~~WIP: Add TimestampNTZType support for casts and unix_timestamp~~ feat: Add TimestampNTZType support for casts and unix_timestamp [WIP] Jan 23, 2026

andygrove added 2 commits January 23, 2026 09:55

docs

08e1d99

Merge remote-tracking branch 'apache/main' into feat/timestamp-ntz-su…

072e70f

…pport

andygrove and others added 5 commits January 23, 2026 11:04

style: run prettier on compatibility.md

1009002

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

style: run cargo fmt

2e468e6

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Merge remote-tracking branch 'apache/main' into feat/timestamp-ntz-su…

3f1583f

…pport

andygrove changed the title ~~feat: Add TimestampNTZType support for casts and unix_timestamp [WIP]~~ feat: Add TimestampNTZType support for casts and unix_timestamp Jan 24, 2026

andygrove marked this pull request as ready for review January 24, 2026 04:36

andygrove mentioned this pull request Jan 25, 2026

[Bug] hour/minute/second expressions incorrectly apply timezone conversion to TimestampNTZ inputs #3180

Open

Merge remote-tracking branch 'origin/main' into feat/timestamp-ntz-su…

38b6c29

…pport # Conflicts: # docs/source/user-guide/latest/compatibility.md

andygrove added 2 commits February 13, 2026 13:55

upmerge

0942101

fix merge conflict

a90a6e9

andygrove marked this pull request as draft February 13, 2026 21:31

andygrove added 3 commits February 13, 2026 14:37

Merge remote-tracking branch 'apache/main' into feat/timestamp-ntz-su…

b13e0dc

…pport

upmerge

601ebe0

merge: resolve conflicts with apache/main and fix clippy warning

6f84317

Merge remote-tracking branch 'apache/main' into feat/timestamp-ntz-su…

c3241f5

…pport

andygrove marked this pull request as ready for review April 21, 2026 12:37

andygrove mentioned this pull request Apr 21, 2026

fix: cast to and from timestamp_ntz #4008

Merged

andygrove closed this Apr 22, 2026

parthchandra mentioned this pull request Apr 23, 2026

feat: Add TimestampNTZType support for unix_timestamp #4039

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add TimestampNTZType support for casts and unix_timestamp#3253

feat: Add TimestampNTZType support for casts and unix_timestamp#3253
andygrove wants to merge 19 commits intoapache:mainfrom
andygrove:feat/timestamp-ntz-support

andygrove commented Jan 23, 2026

Uh oh!

codecov-commenter commented Jan 23, 2026 •

edited

Loading

Uh oh!

parthchandra commented Feb 6, 2026

Uh oh!

andygrove commented Feb 13, 2026

Uh oh!

andygrove commented Feb 13, 2026

Uh oh!

andygrove commented Apr 2, 2026

Uh oh!

parthchandra commented Apr 22, 2026

Uh oh!

andygrove commented Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

andygrove commented Jan 23, 2026

Summary

Test plan

Uh oh!

codecov-commenter commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

parthchandra commented Feb 6, 2026

Uh oh!

andygrove commented Feb 13, 2026

Uh oh!

andygrove commented Feb 13, 2026

Uh oh!

andygrove commented Apr 2, 2026

Uh oh!

parthchandra commented Apr 22, 2026

Uh oh!

andygrove commented Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov-commenter commented Jan 23, 2026 •

edited

Loading